feat: quit checkpoint engine when error occurs #40
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
specture724
requested review from
Copilot and
weixiao-huang
and removed request for
Copilot
There was a problem hiding this comment.
Pull Request Overview
This PR implements error handling in the checkpoint worker process to ensure graceful shutdown when errors occur during weight updates. The changes propagate exceptions from worker processes back to the parameter server, allowing coordinated error handling across all distributed ranks.
Key Changes:
- Worker processes now send exception objects to the parameter server when
run()fails, instead of silently failing - Parameter server synchronizes error states across all ranks before continuing, ensuring no rank proceeds if any rank encounters an error
- Process group cleanup moved to a
finallyblock to ensure proper resource cleanup even when errors occur
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| checkpoint_engine/worker.py | Added try-except block to catch exceptions during weight updates and send them back to the parameter server |
| checkpoint_engine/ps.py | Modified error handling to collect responses from all ranks and raise error if any rank failed; moved cleanup to finally block |
| tests/test_error_quit.py | Added integration test that verifies proper error propagation and process termination when worker process encounters runtime errors |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
specture724
changed the title
…nt-engine into fix/quit-when-error
…nt-engine into fix/quit-when-error
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolve #38
send an
Exceptionto PS whenupdate_weights_from_ipcexecuterunfailed._update_per_bucketneeds to ensure all processes not to get anExceptionbefore continuing next updating iteration